Dishwasher with cloud connected cameras

ABSTRACT

Systems and methods for monitoring dish ware position and modifying the wash and dry cycle of a dishwasher by capturing and analyzing image data utilizing a plurality of cloud-connected cameras.

BACKGROUND

Modern dishwashers have many systems and integrated sensors that aredesigned to provide superior cleaning of dirty dishes during a washcycle. Some dishwashers are equipped with various sensors that arecapable of determining the position of items on the various dishwasherracks and controlling a wash cycle based on those positions. Somedishwashers have a predefined wash function and trajectory where wateris sprayed throughout the racks regardless of their content. The energyconsumed by the dishwashing appliance is therefore directed to all areasof the interior dishwasher racks regardless of the content of thedishwasher.

Some prior art devices use limited vision systems to approximate orestimate the dishware disposed on the racks within a dishwasher cavityand thus tailor the dishwasher cycle accordingly. However these priorart systems are limited in their ability to detect dish ware orcontainers that have moved or flipped over during a wash cycle, which isa particular problem with light plastic containers. When thesecontainers overturn they tend to fill with wash water and thus don't getcompletely clean, and hold soapy or soiled wash water inside thedishwasher cavity during the remainder of the wash cycle. This problemis particularly acute in modern dishwasher systems that operate withreduced water volumes, since the water held within the overturnedcontainer is needed for system operation.

Thus there is a need in the art for a dishwasher have dish ware sensingcapability within the dishwasher cavity that has the ability to detectif containers have flipped during the wash cycle as a result of jetimpingement. In these instances a user may then be alerted to adjust theout of place containers so they do not hold water that is needed by thesystem.

Additionally, there is a further need in the art for a dishwasher havedish ware sensing capability within the dishwasher cavity that has theability to modify detergent dosing or metering in a bulk dispense setupbased on the dish ware load size and arrangement.

SUMMARY

The present disclosure is related to systems and methods for sensing andmonitoring the location and arrangement of dish ware on the variousracks of a dishwasher appliance. The system disclosed herein utilizes aplurality of sensors disposed at a plurality of locations within thedishwasher cavity capture images or imaging data and transmit that datato a remote server or cloud-based computing platform for analysis. Invarious embodiments the imaging data collected is analyzed by ananalysis system that may include a deep learning algorithm to detect theposition of dish ware on the various racks of the dishwasher. In someembodiments the sensors employed may be cameras, RF sensors (radiofrequency transmitters and detectors) and LIDAR sensors (light detectionand ranging) to detect and determine dish ware location and arrangement.

In various embodiments, the systems and methods disclosed herein providea system that analyzes images or image data from a plurality of sensorsarrayed within a dishwasher cavity to determine the initial location ofthe dish ware therein and adjust or modify a dish washing cycle or otherwash/rinse/dry variable depending upon the location of various items. Inother embodiments, image data may be taken at the initiation of a washcycle and then compared with imaging data sensed later in the dishwashing cycle to determine an out-of-place or overturned piece of dishware.

In other embodiments, the system and methods disclosed herein may beused to analyze the locations and orientations of dish ware in quadrantsor sections of a dish washer cavity and then adjust wash/rinse/dry cyclevariables according to the items detected in a particular section orquadrant. In some aspects the sensors employed may be capable ofdetecting the types of materials of variously located items of dish wareand once analyzed, the system may alter the wash cycle of the dishwasheraccordingly.

In other aspects of the disclosure the system may prompt a user to stopthe wash cycle of the dishwasher to re-orient an out of place oroverturned item that is holding water and thus reducing theeffectiveness of the wash cycle. In other embodiments the system may usea remotely obtained image provided by a user via a communicationsinterface to conduct an analysis of the dishwasher load and modify thewash cycle, or prompt a user to take corrective action.

As used herein for purposes of the present disclosure, the term“appliance” should be understood to be generally synonymous with andinclude any device that consumes electrical power and can be connectedto an electrical circuit, for example one used in a residential orcommercial setting to accomplish work. The appliances referred to hereinmay include a plurality of electrically operated components powered bythe circuit and further may include a processor or processors thatoperate the appliance.

The term “dishwasher” should be understood to be an appliance thatincludes a “processor” as that term is understood below, for operatingand controlling the various electrical, electro-mechanical andmechanical components included in the dishwasher, and a door enclosingan interior cavity into which dish ware may be placed for cleaning. Adishwasher may also comprise a plurality of racks disposed within thecavity as well as a water jet system for directing cleaning water ontodish ware and a soap and rinse aid dispensing system for supplying soapand rinse aid to the cavity during a wash cycle.

The terms “images” or alternatively “imaging data” should be understoodto refer to all types of digital or analog electronic data, and/ordigital or analog electrical signals that may be utilized either beforeor after additional signal processing, to interpret the presence and/ortype of dish ware or other object present in an area being sensed.Images or imaging data may refer to all types of signals, analog ordigital produced and provided by imaging sensors such as opticalsensors, LIDAR sensors, RF sensors, or any other type of electrical orelectronic sensor used to detect and analyze objects in space.

The term “Internet” or synonymously “Internet of things” refers to theglobal computer network providing a variety of information andcommunication facilities, consisting of interconnected networks usingstandardized communication protocols. The appliances, controllers andprocessors referred to herein may be operatively connected to theInternet.

As used herein for purposes of the present disclosure, the term“wireless communication” generally describes apparatus and systemsrelating to the wireless transmission of a signal. Any of a wide varietyof wireless transmission devices and communications protocols may beemployed in the system of the invention, including analog and digitaltransmission systems. Exemplary but non-limiting wireless transmittersthat may form a part of the invention include radio transmitters,cellular transmitters, LTE and LTE advanced systems, ZigBee™, Wi-Fi, andBluetooth transmitters. Additionally, a plurality of wireless networkand transmission systems may be employed without departing from thescope of the invention, including, but not limited to, wireless personalarea networks, local area networks, mesh networks, metropolitan area andglobal area networks.

The term “processor” or alternatively “controller” is used hereingenerally to describe various apparatus relating to the operation of oneor more computers, web servers, or databases. A processor can beimplemented in numerous ways (e.g., such as with dedicated hardware) toperform various functions discussed herein. A “processor” is one exampleof a controller which employs one or more microprocessors that may beprogrammed using software instructions (e.g., microcode) to performvarious functions discussed herein. A controller may be implemented withor without employing a processor, and also may be implemented as acombination of dedicated hardware to perform some functions and aprocessor (e.g., one or more programmed microprocessors and associatedcircuitry) to perform other functions. Examples of controller componentsthat may be employed in various embodiments of the present disclosureinclude, but are not limited to, conventional microprocessors,application specific integrated circuits (ASICs), and field-programmablegate arrays (FPGAs).

In various implementations, a processor or controller may be associatedwith one or more storage media (generically referred to herein as“memory,” e.g., volatile and non-volatile computer memory such as RAM,PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks,magnetic tape, etc.). In some implementations, the storage media may beencoded with one or more programs that, when executed on one or moreprocessors and/or controllers, perform at least some of the functionsdiscussed herein. Various storage media may be fixed within a processoror controller or may be transportable, such that the one or moreprograms stored thereon can be loaded into a processor or controller soas to implement various aspects of the present disclosure discussedherein. The terms “program” or “computer program” or “instructions” areused herein in a generic sense to refer to any type of computer code(e.g., software or microcode) that can be employed to program one ormore processors or controllers.

The term “user interface” as used herein refers to an interface betweena user or operator and one or more devices that enables interactionbetween the user and the device(s). Examples of user interfaces that maybe employed in various implementations of the present disclosureinclude, but are not limited to, switches, potentiometers, buttons,dials, sliders, a mouse, keyboard, keypad, various types of gamecontrollers (e.g., joysticks), track balls, display screens, varioustypes of graphical user interfaces (GUIs), smartphones, watches,tablets, personal computing platforms, touch screens, microphones andother types of sensors that may receive some form of human-generatedstimulus and generate a signal in response thereto. Furthermore, userinterfaces can encompass interactive web pages and other user prompts,whether provided on stand alone computing platforms or mobile devices.

The terms “communications interface” or alternatively “communicationslink” are generally meant to include in digital or other communicationwith any other part of the system via a wireless or wired communicationprotocol. A communication interface may be between two devices orcomponents and may be accomplished by a separate networking system.Communication interfaces may be provided to transfer data between a webserver, a database, a computer, a mobile or handheld device, or anyother control system, a consumer operated external device, a wirelesslocal area network (WLAN), or any other communication system. Thecommunication links disclosed and described in this specification may beintegrated within various system components or alternatively may beseparate electronic systems.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein. In particular, allcombinations of claimed subject matter appearing at the end of thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein. It should also be appreciated that terminologyexplicitly employed herein that also may appear in any disclosureincorporated by reference should be accorded a meaning most consistentwith the particular concepts disclosed herein.

Before explaining exemplary embodiments consistent with the presentdisclosure in detail, it is to be understood that the disclosure is notlimited in its application to the details of constructions and to thearrangements set forth in the following description or illustrated inthe drawings. The disclosure is capable of embodiments in addition tothose described and is capable of being practiced and carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein, as well as in the abstract, are for thepurpose of description only and should not be regarded as limiting.

The accompanying drawings, which are incorporated and form a part of thespecification illustrate exemplary, but non-limiting, embodiments of thedisclosure, and together with the description, serve to explain theprinciples of the disclosure.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are part of the inventivesubject matter disclosed herein. In particular, all combinations ofclaimed subject matter appearing at the end of this disclosure arecontemplated as being part of the inventive subject matter disclosedherein. It should also be appreciated that terminology explicitlyemployed herein that also may appear in any disclosure incorporated byreference should be accorded a meaning most consistent with theparticular concepts disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. The drawings are not necessarilyto scale. Emphasis is instead generally placed upon illustrating theprinciples of the disclosure, wherein;

FIG. 1 is an isometric diagram of a dishwasher in accordance withvarious embodiments; and

FIG. 2 is a side view of an open dishwasher in accordance with variousembodiments;

FIG. 3 is a side view of an open dishwasher in accordance with variousembodiments;

FIG. 4 is an isometric view of an open dishwasher in accordance withvarious embodiments;

FIG. 5 is an isometric view of an open dishwasher in accordance withvarious embodiments;

FIG. 6 is an isometric view of an open dishwasher in accordance withvarious embodiments;

FIG. 7 is an isometric view of an open dishwasher in accordance withvarious embodiments; and

FIG. 8 is an isometric view of an open dishwasher in accordance withvarious embodiments;

DETAILED DESCRIPTION

Referring to drawing FIGS. 1-4, and in accordance with various aspectsand embodiments of the invention, a system 10 for sensing and monitoringthe location and arrangement of dish ware 2, 3 on the various racks 102of a dishwasher 100 is described. In various embodiments the dishwasher100 in which system 10 is implemented may include a controller 200integral to dishwasher 100 that operates dishwasher 100 by controllingand monitoring the various components thereof, and of system 10, as willbe described in detail herein below.

FIG. 1 illustrates an exemplary dishwasher 100 hardware environment forimplementing system 10 for sensing and monitoring the location andarrangement of dish ware 2, 3. The dishwasher 100 may include acontroller 200, a processor or processors 202 and memory 204. Dishwasher100 may further comprise a plurality of signal outputs 210 and signalinputs 220 that may be operatively connected to a plurality ofdishwasher 100 components to monitor and direct system 10 operation.Furthermore, in some embodiments controller 200 may include a wirelessor hard-wired communications interface 230 that enables controller 200to communicate with external devices or communications networks such asthe internet or a remotely located server 260, that may be integratedinto system 10.

Additionally, controller 200 may be equipped with an operator or userinterface 240 to provide audible or visual feedback to a user as well asenable a user the ability to provide instructions or commands tocontroller 200 and thus dishwasher 100. Exemplary but non-limiting userinterfaces that may be employed include a mouse, keypads, touch-screens,keyboards, switches, buttons and/or touch pads. Any user interface maybe employed for use in the invention without departing from the scopethereof. It will be understood that FIG. 1 constitutes, in somerespects, an abstraction and that the actual organization of thecomponents of dishwasher 100 and controller 200 may be more complex thanillustrated.

The processor 202 may be any hardware device capable of executinginstructions stored in memory 204 or data storage 206 or otherwiseprocessing data. As such, the processor may include a microprocessor,field programmable gate array (FPGA), application-specific integratedcircuit (ASIC), or other similar devices.

The memory 204 may include various memories such as, for example L1, L2,or L3 cache or system memory. As such, the memory 204 may include staticrandom access memory (SRAM), dynamic RAM (DRAM), flash memory, read onlymemory (ROM), or other similar memory devices. It will be apparent that,in embodiments where the processor includes one or more ASICs (or otherprocessing devices) that implement one or more of the functionsdescribed herein in hardware, the software described as corresponding tosuch functionality in other embodiments may be omitted.

The user interface 240 may include one or more devices for enablingcommunication with a user such as an administrator. For example, theuser interface 240 may include a display, a mouse, and a keyboard forreceiving user commands. In some embodiments, the user interface 240 mayinclude a command line interface or graphical user interface that may bepresented to a remote terminal via the communication interface 230.

The communications interface 230 may include one or more devices forenabling communication with other hardware devices. For example, thecommunication interface 230 may include a network interface card (NIC)configured to communicate according to the Ethernet protocol.Additionally, the communication interface 230 may implement a TCP/IPstack for communication according to the TCP/IP protocols. Variousalternative or additional hardware or configurations for thecommunication interface 230 will be apparent.

The storage 206 may include one or more machine-readable storage mediasuch as read-only memory (ROM), random-access memory (RAM), magneticdisk storage media, optical storage media, flash-memory devices, orsimilar storage media. In various embodiments, the storage 206 may storeinstructions for execution by the processor 202 or data upon with theprocessor 202 may operate. For example, the storage 206 may store a baseoperating system for controlling various basic operations of thehardware. Other instruction sets may also be stored in storage 206 forexecuting various functions of system 10, in accordance with theembodiments detailed below.

It will be apparent that various information described as stored in thestorage 206 may be additionally or alternatively stored in the memory204. In this respect, the memory 204 may also be considered toconstitute a “storage device” and the storage 206 may be considered a“memory.” Various other arrangements will be apparent. Further, thememory 204 and storage 206 may both be considered to be “non-transitorymachine-readable media.” As used herein, the term “non-transitory” willbe understood to exclude transitory signals but to include all forms ofstorage, including both volatile and non-volatile memories.

While the controller 200 is shown as including one of each describedcomponent, the various components may be duplicated in variousembodiments. For example, the processor 202 may include multiplemicroprocessors that are configured to independently execute the methodsdescribed herein or are configured to perform steps or subroutines ofthe methods described herein such that the multiple processors cooperateto achieve the functionality described herein. Further, where thecontroller 200 is implemented in a cloud computing system, the varioushardware components may belong to separate physical systems. Forexample, the processor 202 may include a first processor in a firstserver and a second processor in a second server.

Referring now to FIGS. 1-4, in accordance with some embodiments,dishwasher 100 may include a hinged door 110 that opens to exposedishwasher 100 cavity 120, into which dish ware 2, 3, is placed forcleaning. Cavity 120 is defined by door 120, a pair of spaced side walls122, a rear wall 124, and top 126 and bottom wall 128. Dishwasher 100may also include rack 140, or a plurality thereof, that are spacedwithin cavity 120 and on which dish ware 2, 3 is disposed for cleaning.As is known in the art racks 140 may include wheels or runners such thatthey slide outwardly from cavity 140 when door 110 is open for access toload and remove dish ware 2, 3. Additionally, dishwasher 100 istypically equipped with a liquid circulation system (not shown) forintroducing, directing, and circulation liquid and wash aids andsolutions such as detergents, rinse aid and the like, throughout cavity110 during a wash cycle. The liquid circulation system is typicallyprovided with a plurality of sprayers, movable or circulating spray armsor the like for directing wash and rinse solution onto dish ware 2, 3.

In one non-limiting exemplary embodiment for purposes of illustration inthis specification, dishwasher 100 includes at least one dish waresensor 300, or a plurality thereof, arranged in and secured at aplurality of locations in cavity 120. Sensors 300 may in variousembodiments of the invention be optical sensors, cameras, RF (radiofrequency) sensors, LIDAR modules, infrared sensors or other equivalentsensors capable of detecting the presence and location of dish ware 2,3. In some embodiments a plurality of camera sensors 300 are secured ata plurality of locations throughout cavity 120 of dishwasher 100 todetect and monitor the location of dish ware disposed on racks 140.Sensors 300 include an output 302 operatively coupled to an input 220 ofcontroller 200, the output 302 being representative of the view of dishware 2, 3, from sensor 300 for further processing. In variousembodiments processor 200 provides image data captured from sensors 300to remote server 260 through operation of wireless communicationsinterface 230 for additional processing.

As best depicted in FIGS. 2-4 and in accordance with variousnon-limiting embodiments cameras 300 may be arranged to provide a viewof dish ware 2, 3 on racks 140 from above, below, and on both sidesthereof. For example, cameras or sensors 300 are secured at multiplelocations along side walls 122, top 126, and bottom 128. Sensors 300 mayalso be placed on door 110 interior, to obtain a view of dish ware 2, 3from the front of dishwasher 100. Furthermore, as best depicted in FIGS.2 and 3, a sensor 300 may be disposed proximate the top of dishwasher100, on the exterior thereof outside cavity 120, such that an image maybe taken of the top rack 140 and bottom rack 140 when racks 140 areloaded but extended out of cavity 120. In this embodiment the naturallight provided by the environment or room interior in which dishwasher100 is located may be utilized to obtain an image of dish ware 2, 3,placed on racks 140 prior to closing door 110. This feature of theinvention provides processor 200 with a “before’ image of dish ware 2, 3that can be compared to a subsequent image to determine whether dishware 2, 3 has moved or become dislodged due to jet impingement.

A plurality of lights 320 may be mounted at a plurality of locations ondishwasher sides 122, top 126, back 124 and bottom 128 to provide lightduring operation of dishwasher 100, since once door 110 is closed noambient light is available for sensors 300 that require it. In someembodiments, a single diffuse light source may be secured in cavity 120,for example recessed in top 126 thereof. In these embodiments, multipleimages can be procured by cameras 300 during dishwasher 100 operationand then analyzed by system 10, as detailed herein below.

Referring now to FIGS. 6 and 7 and in accordance with variousembodiments a horizontally mounted gantry 340 may be disposed along rearwall 124. A sensor 300, or a plurality thereof may be mounted on gantry340 whereby gantry 340 moves sensor 300 horizontally acrosssubstantially the entire width of rear wall 124 during dishwasheroperation so that a plurality of images of dish ware 2, 3 can be taken.In some embodiments a pair of sensors 300 may be mounted on gantry 340,one looking upwardly at upper rack 140 and one looking downwardly atlower rack 140 so that images may be take of all dish ware 2, 3 on eachrack. In this fashion essentially all dish ware 2, 3 present on racks140 can be monitored by sensors 300 mounted on gantry 340 duringoperation of dishwasher 100. The movement of sensors 300 along gantry340 may be controlled by an output 210 of processor 200, therebypermitting timed control of images taken by sensors 300 duringdishwasher 100 operation. Gantry 340 may be operated by a servomotor,linear actuator, or equivalent mechanism without departing from thescope of the present invention. Additionally and alternatively gantry340 may be mounted along side 122 or rear 124 walls in a verticalorientation such that sensors are moved upwardly and downwardly whengantry 340 is operated.

In a yet further embodiment and in accordance with some aspects of theinvention a LIDAR (Laser imaging detection and ranging) imaging module380 or RF (radio frequency) sensor module 390 may be mounted or securedto gantry 340 to monitor dish ware 2, 3 during dishwasher 100 operation.RF sensor 390 may be mounted or secured to a fixed location withincavity 120, or mounted on gantry 340. LIDAR module 380 includes anoutput 382 representative of an image of dishware 2, 3, in dishwasher100 that is operatively coupled to an input 220 of processor 200. Invarious embodiments processor 200 provides image data captured fromLIDAR module 380 to remote server 260 through operation of wirelesscommunications interface 230 for additional processing. Similarly, RFsensor 390 includes an output 392 representative of an image of dishware2, 3, in dishwasher 100 that is operatively coupled to an input 220 ofprocessor 200. Processor 200 provides image data captured from RF sensor390 to remote server 260 through operation of wireless communicationsinterface 230 for additional processing.

In operation, system 10 is capable of capturing an image of dish ware 2,3 contained within a loaded dishwasher 100 prior to beginning a washcycle and comparing it with subsequent images captured during the cycle.In some aspects and embodiments processor 200 may provide an output 210to command sensors 300 to capture an image of dish ware 2, 3. This imagecapture command may be provided just prior to the initiation of a washcycle, or even immediately following the initiation of a wash cycle. Inthose embodiments where a sensor 300 is mounted externally of dishwasher100 cavity 120 as depicted in FIGS. 2 and 3, a user may be able toinitiate an image capture of racks 140 by selecting an image capturecommand through appropriate selections on operator interface 240.Alternatively, when a user selects a wash cycle processor 200 may prompta user via operator interface 240 to extend racks 140 to capture imagesof dish ware 2, 3, arrange there on prior to closing door 110 andinitiating a wash cycle.

In various embodiments the images of dish ware 2, 3, captured by sensors300 is transmitted via communications interface 230 to a remote serversuch as a cloud server 260 or other remote computing platform for imageanalysis. The images provided for an individual dish washer 100 can thenbe analyzed by a deep learning algorithm to classify various items ofdish ware and log their locations in a load analysis. Once the loadanalysis has taken place, server 260 can then relay wash cycleadjustment instructions to dishwasher 100 via communications interface230. In an exemplary but non-limiting embodiment, where sensors 300capture images indicating a preponderance of light weight plastic dishware 2 on a given rack, or even in a predetermined area or quadrant ofdishwasher cavity 120, the analysis system may provide instructionsthrough server 260 to operate the spray nozzles directing cleaningsolution to that area at a reduced jet pressure or velocity, ordirection during the various wash and rinse cycles for a givendishwasher cycle. In these embodiments system 10 can customize a washcycle based on the type of dish ware 2, 3 placed on racks 140 therebyavoiding or at least reducing the problematic overturning of lightweight re-sealable plastic containers such commonly used to store fooditems.

In some aspects and embodiments the images or imaging data provided bysensors 300 for a given dishwasher 100 load can be analyzed by sectors,sections or quadrants of the cavity 120 of dishwasher 100, dependingupon the location and arrangement of the plurality of sensors 300mounted therein. In these embodiments the dish washer wash and rinsecycles may be modified during operation based upon the contents andlocation of dish wares 2, 3 in a given section or quadrant. In someembodiments the two-dimensional images captured by sensors 300 can beanalyzed using an image analysis system such as a deep learningalgorithm to create an edge profile of each piece of dish ware 2 locatedin cavity 120 and thereby provide an approximate location and anestimate of what type of dish ware 2, 3 is present. For example, certainedge profiles will positively identify a piece of dish ware as a glassbowl or dish while others will positively identify the dish ware 2 as aplastic container. Using the edge profiles of the various items incavity 120 the analysis system may provide cycle modificationinstructions to dishwasher 100 processor 200 to modify the wash, rinseor dry cycles, select a particular cycle or action, or even add or omita specified amount of detergent or rinse aid.

In some additional embodiments, the images analyzed by the analysissystem during operation can provide edge profiles and position data to adeep learning algorithm that can identify a “flipped” or overturnedcontainer. These overturned items are particularly problematic in moderndishwashers 100 because they collect a volume of water that is neededfor proper operation of the dishwasher 100 wash and rinse cycles. A deeplearning algorithm can compare dishwasher 100 load images from sensors300 taken just prior to wash cycle initiation with subsequent imagestaken during wash cycle operation. By comparing these images, thelearning algorithm can discover when an item has been overturned andaccordingly initiate instructions through server 260 to processor 200 toalert the user via operator interface 240 to take corrective action.Processor 200 may stop or suspend the wash cycle and then provide avisual or audible alarm via operator interface 240 to alert a user toopen door 110 and rearrange the offending item before the cycle ispermitted to continue. This feature of the invention is particularlyuseful when washing a rack or load of light weight items that tend tooverturn and hold water.

In some embodiments of the invention where RF sensors 300 are utilizedto capture dish ware 2, 3, image data, an RF sensor 300 module having aplurality of antenna pairs may be positioned and secured either on agantry 340 or proximate one of the four corners of each dishwasher 100rack 140. RF sensors 300 may then be calibrated by processor 200 byinstructing RF module 300 to transmit and receive RF signals while racks140 are empty, thereby creating a profile for an empty dishwasher 100.Once calibrated, RF modules 300 can then be used to detect and transmitthe locations of various items placed on racks 140, since the data fromthe empty dishwasher can be filtered out of the received RF signals. Asbefore, the data captured by RF modules 300 can be transmitted viacommunications interface 230 to a cloud server 260 for analysis with adeep learning algorithm. In some aspects and embodiments RF sensors 300may utilize image processing using a coordinate system, for example aCartesian or polar coordinate system, to determine and track exact dishware 2, 3, position in cavity 120. Accordingly, the imaging dataprovided by RF sensor 300 embodiments can be analyzed by comparing thecoordinates of the various objects identified over a predetermined timespan, or even continuously.

In some aspects and embodiments RF sensors 300 may provide imaging datathat can be analyzed to determine the type of dish ware 2, 3 present inracks 140. For example, RF signals detected from the presence of glassware are substantially different that those received from the presenceof plastic ware. Accordingly, the RF sensor 300 imaging data can beanalyzed in remote server 260 to determine both a location and a dishware material and use this data to adjust the wash cycle. Alternatively,and similar to some embodiments discussed herein above, a deep learningalgorithm can compare dishwasher 100 load images (or imaging data) fromRF sensors 300 taken just prior to wash cycle initiation with subsequentimages taken during wash cycle operation. By comparing these “before”and “after” images, the learning algorithm can discover when an item hasbeen overturned and accordingly initiate instructions through server 260to processor 200 to alert the user via operator interface 240 to takecorrective action. Processor 200 may then stop or suspend the wash cycleand then provide a visual or audible alarm via operator interface 240 toalert a user to open door 110 and rearrange the offending item beforethe cycle is permitted to continue.

In various aspects and embodiments sensors 300 may comprise a LIDARarray or arrays 300 for obtaining and providing imaging data for furtheranalysis. In these embodiments the LIDAR sensors 300 may be mounted orsecured in predetermined locations in dishwasher cavity 120 to obtain acomplete image thereof or alternatively they may be mounted to a gantry340 secured on back wall 124 that moves the LIDAR 300 sensorhorizontally between, for example, an upper and lower rack 140. LIDARsensors 300 can provide imaging data to processor 200 and remote server260 representative of the surfaces of dish ware 2, 3 present in cavity120. LIDAR sensors 300 capture imaging data that correspond to thesurface characteristics of the dish ware 2, 3 such that an algorithm candetermine what surface of a give item should be oriented upwardly ordownwardly in a rack 140. Based on this imaging data from LIDAR sensor300, the learning algorithm can discover when an item has beenoverturned and accordingly initiate instructions through server 260 toprocessor 200 to alert the user via operator interface 240 to takecorrective action. Processor 200 may stop or suspend the wash cycle andthen provide a visual or audible alarm via operator interface 240 toalert a user to open door 110 and rearrange the offending item beforethe cycle is permitted to continue.

In a yet further aspect and embodiment of the invention, a user canprovide a remotely obtained image to processor 200 via communicationsinterface 230 for analysis. In this embodiment an image or pluralitythereof taken, for example, by a cell phone or other device capable ofcapturing and transmitting digital images to processor 200 and thusserver 260. These remotely obtained images can then be analyzed asbefore utilizing a deep learning algorithm to determine whether anycycle adjustment or other action need be taken, either automaticallythrough operation of processor 200 or through user action occasioned bya prompt via operator interface 240.

While a variety of inventive embodiments have been described andillustrated herein, those of ordinary skill in the art will understandthat a variety of other methods, systems, and/or structures forperforming the function and/or obtaining the results, and/or one or moreof the advantages described herein are possible, and further understandthat each of such variations and/or modifications is within the scope ofthe inventive embodiments described herein. Those skilled in the artwill understand that all parameters, dimensions, materials, andconfigurations described herein are meant to be exemplary and that theactual parameters, dimensions, materials, and/or configurations willdepend upon the specific application or applications for which theinventive teachings is/are used. Those skilled in the art willrecognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific inventive embodimentsdescribed herein. It is, therefore, to be understood that the foregoingembodiments are presented by way of example only and that, within thescope of the appended claims and equivalents thereto, inventiveembodiments may be practiced otherwise than as specifically describedand claimed. Inventive embodiments of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the inventive scope of thepresent disclosure.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

It should also be understood that, unless clearly indicated to thecontrary, in any methods claimed herein that include more than one stepor act, the order of the steps or acts of the method is not necessarilylimited to the order in which the steps or acts of the method arerecited.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03. It should be understoodthat certain expressions and reference signs used in the claims pursuantto Rule 6.2(b) of the Patent Cooperation Treaty (“PCT”) do not limit thescope.

What is claimed is:
 1. A system for sensing and monitoring the locationand arrangement of dish ware on at least one rack inside the cavity of adishwasher to modify a wash cycle comprising: a processor andconcomitant data memory, said processor having a plurality of inputs andoutputs for receiving and providing electrical signals to a plurality ofelectrical components of said dishwasher; a user interface operativelycoupled to said processor for accepting inputs from a user and providingdishwasher cycle selections to a user; a communications interfaceoperatively coupled to said processor that enables electroniccommunications from said processor to a remote device; a sensor disposedwithin said dishwasher cavity for capturing images of said dish ware,said sensor having a data imaging output representative of said dishware operatively coupled to an input of said processor; a remote serverin signal communication with said processor, said server having animaging data analysis system for analyzing said images; and wherein saidanalysis system determines the position and composition of dish warebased on said imaging data and provides wash cycle instructions to saidprocessor.
 2. The system of claim 1 wherein said analysis system isfurther configured to: identify an overturned item of dish ware; andprovide instructions to said processor to pause said wash cycle andalert a user to reposition said overturned item.
 3. The system of claim2 comprising: a plurality of sensors disposed within said dishwashercavity for capturing images of said dish ware in a plurality ofsections.
 4. The system of claim 3 wherein said sensors comprise aplurality of cameras.
 5. The system of claim 3 wherein said sensorscomprise radio frequency sensors.
 6. The system of claim 3 wherein saidsensors comprise LIDAR sensors.
 7. The system of claim 2 wherein saidanalysis system is further configured to: identify a predetermined typeof dish ware by material; and provide instructions to said processor tomodify said wash cycle based on said identified dish ware material. 8.The system of claim 1 comprising: a gantry disposed with said dishwashercavity having said sensor secured to a movable portion thereof, whereinsaid gantry moves said at least one sensor to a plurality of locationsto capture images prior to and during said wash cycle.
 9. The system ofclaim 8 wherein said sensor is a LIDAR sensor.
 10. The system of claim 8wherein said sensor is an RF sensor.
 11. The system of claim 8 whereinsaid gantry is disposed horizontally in said dishwasher cavity.
 12. Thesystem of claim 1 wherein said analysis system comprises a deep learningalgorithm.
 13. The system of claim 1 comprising: a sensor positionedexterior to said dishwasher cavity for capturing images of dish waredisposed on said racks prior to initiation of a wash cycle.
 14. Thesystem of claim 1 wherein said processor accepts images of dish waredisposed in said cavity from a remote image processing device.
 15. Amethod of sensing and monitoring the location and arrangement of dishware on at least one rack inside the cavity of a dishwasher, saiddishwasher having a processor, a plurality of inputs and outputs forreceiving and providing electrical signals to a plurality of electricalcomponents of said dishwasher, a user interface operatively coupled tosaid processor, a communications interface operatively coupled to saidprocessor that enables electronic communications from said processor toa remote device; a sensor disposed within said dishwasher cavity forcapturing images of said dish ware, and a remote server in signalcommunication with said processor, said server having an imaging dataanalysis system for analyzing said images; comprising: providing ananalysis system for determining the position and composition of dishware based on said imaging data; and providing wash cycle instructionsto said processor.
 16. The method of claim 15 comprising: determiningthe positions of dish ware in said dish washer; monitoring said dishware positions at a plurality of predetermined times throughout theduration of said wash cycle; determining whether an item of dish warehas overturned based on said positions; providing instructions to saidprocessor to alert a user when an item has overturned.
 17. The method ofclaim 15 comprising: determining the edge profiles of dish ware in saiddish washer; monitoring said edge profiles at a plurality ofpredetermined times throughout the duration of said wash cycle;determining whether an item of dish ware has overturned based on saidedge profiles; providing instructions to said processor to alert a userwhen an item has overturned.
 18. The method of claim 15 comprising:determining the material types of dish ware in said dish washer;providing instructions to said processor to modify said wash cycle basedon the material types of dish ware.
 19. The method of claim 15 whereinsaid analysis system comprises: a deep learning algorithm for comparingan image of an arrangement of dishware prior to a wash cycle with aplurality of images of said arrangement of dishware at predeterminedsubsequent times